FACT: An Assessment Framework for Model-Based Program Comprehension Tools

Model-based program comprehension tools are those that use some model of a domain to assist in analysing software, and represent a large proportion of support environments for software maintainers. Such tools have widely differing characteristics that affect their suitability for software maintenance tasks. Existing evaluations of these tools take many forms, often addressing only a small part of the information needed by a maintainer to decide which tool or type of tool to apply. Consequently, comparison can be difficult. Existing approaches can be broadly divided into those that address the nature of the tool (in terms of technical characteristics and ability), and those that address the application of the tool (e.g. the psychological support provided). We present a new framework, applicable to model-based program comprehension tools, that allows a maintainer to analyse systematically the properties of their tool and determine its suitability for a given task. We explain the ten areas the framework covers, explaining why each is important, and demonstrating its application to the Hypothesis-Based Concept Assignment System (HB-CAS). In addition to its application by software maintainers, we anticipate the framework being useful to other researchers as a basis for evaluating and comparing their work with others.